(1) Field of the Invention
The present invention relates to a picture reproducing apparatus capable of receiving coded picture or video data from a network and decoding and reproducing the picture. More specifically, the present invention relates to an encoding/decoding apparatus which includes an encoder for transmitting coded data and a tool constituting an algorithm for decoding the data and a decoder for reconfiguring the received tool as an algorithm and decoding the received coded data in a program download manner. The present invention also relates to a picture encoder for constructing a software download system by transmitting a coding system each time a communication line is connected when transmitting coded picture data to another terminal through the communication line.
(2) Description of the Prior Art
Recently, a picture communication service is realized as a new communication service with the spread of ISDN (Integrated Services Digital Network). Examples are video phones and video conference systems. Also, with the development of mobile transmission networks represented by PHS and FPLMTS, demands on the delicacy, diversity, and portability of services provided are rapidly increasing.
Generally, in systems such as video phones or video conference systems in which picture information is transmitted, the amount of picture information is enormous. Therefore, when the speed and the cost of lines used in transmission are taken into consideration, it is necessary to decrease the amount of picture information to be transmitted by compressing (coding) the picture information before transmitting it.
As coding systems for compressing picture information, JPEG (Joint Photographic coding Experts Group) as a still picture coding system, H. 261 as a moving picture coding system, and MPEG1 and MPEG2 (Moving Picture coding Expert Group) as storage type moving picture coding systems are already internationally standardized. In addition, MPEG4 as a coding system at a very low bit rate of 64 kbps or less is being standardized.
MPEG4 can flexibly cope with various applications and provide an optimum encoding system for each application. Therefore, unlike the existing JPEG, H. 261, MPEG1, and MPEG2 encoding systems in which encoding is performed in accordance with an algorithm, MPEG4 must be an encoding system in which large numbers of tools (transformers, quantizers, inverse transformers, and inverse quantizers or dequantizers) of an encoder are provided and encoding is performed by properly combining these tools.
FIG. 1A is a view showing the format of a data sequence or stream of coded data obtained by encoding (compressing) picture data in accordance with H. 261. Each coded data such as motion vector information, a DCT coefficient, or a quantization step shown in FIG. 1A is picture information encoded (compressed) on the basis of a fixed encoding algorithm in an encoder. A decoder includes a fixed decoding algorithm with respect to this encoding algorithm and decodes each received coded data.
FIG. 1B is a view showing the format of a data sequence of coded data obtained when an algorithm such as MPEG4 encodes (compresses) picture data by using a flexible encoding system. The coded data sequence shown in FIG. 1B consists of coded (compressed) picture information such as motion vector information 2, a transform coefficient 4, motion vector information 6, a transform coefficient 8, and a quantization step 10, and tool information such as a motion compensation tool 1, an inverse transform tool 3, a motion compensation tool 5, an inverse transform tool 7, and a quantization tool 9 for decoding the respective corresponding picture information. Each tool information such as the motion compensation tool 1 can be selected from a plurality of different pieces of tool information, and any arbitrary combination of these pieces of tool information can be selected. Accordingly, the encoder transmits the picture information and the tool information used in the encoding to the decoder, and the decoder decodes the received coded picture information by using the tool information transmitted from the encoder.
Methods of realizing these encoding/decoding processes include a method using dedicated hardware and software and a method which executes appropriate software by using a general-purpose arithmetic unit and a compiler.
The method which realizes the encoding process by using dedicated hardware and software will be described below. FIG. 2 is a block diagram showing the configuration of an encoder which generates the coded data shown in FIG. 1A in accordance with H. 261. Referring to FIG. 2, this encoder includes an encoding controller 11 for controlling encoding, a transformer 12 for calculating DCTs, a quantizer 13 for quantizing the coefficient transformed by the transformer 12, an inverse quantizer 14 for inversely quantizing the coefficient quantized by the quantizer 13, an inverse transformer 15 for calculating inverse DCTs, a memory 16, and a loop filter 17. The memory 16 has a motion compensation variable delaying function used in motion compensation inter-frame prediction. The filter 17 is a loop filter which can be turned on or off for each macroblock.
In realizing an encoding algorithm for generating the coded data shown in FIG. 1A by using dedicated hardware and software, the functions of tools constituting this algorithm are achieved by dedicated hardware and software, i.e., the encoding controller 11, the transformer 12,. the quantizer 13, the inverse quantizer 14, the inverse transformer 15, the memory 16 having the motion compensation delaying function, and the loop filter 17. FIG. 3 is a block diagram showing the configuration of a decoder for decoding coded data encoded in accordance with H. 261. This decoder shares the components of the encoder shown in FIG. 2. The same reference numerals as in FIG. 2 denote the same parts in FIG. 3. That is, the decoder shown in FIG. 3 includes the inverse quantizer 14, the inverse transformer 15, the memory 16 having the motion compensation variable delaying function, and the loop filter 17.
The coded data encoded by the encoder shown in FIG. 2 is inversely quantized by the inverse quantizer 14 and decoded by calculating inverse DCTs by the inverse transformer 15. The memory 16 and the loop filter 17 are used in decoding motion compensated predictive coded data.
To process several different algorithms by using the system which performs encoding by using a fixed algorithm such as H. 261, dedicated hardware and software must be provided for each algorithm. FIG. 4 is a block diagram showing the structure of an encoder which encodes moving pictures in accordance with H. 261 and encodes still pictures in accordance with JPEG. As an example, when one terminal encodes moving pictures and still pictures in accordance with H. 261 and JPEG, respectively, the configuration of the encoder is as shown in FIG. 4. That is, the encoder includes both an H. 261 encoder 20 and a JPEG encoder 21. In FIG. 4, the H. 261 encoder 20 and the JPEG encoder 21 receive moving picture data and still picture data, respectively, and output respective coded data as compressed data.
To realize an algorithm for generating the coded data shown in FIG. 1B by using dedicated hardware and software, an encoder for realizing this algorithm is achieved by replacing the configuration of a circuit block 18 of the encoder shown in FIG. 2 with a configuration shown in FIG. 5. That is, this encoder includes a plurality of different tools in each of the transformer 12, the quantizer 13, the inverse quantizer 14, and the inverse transformer 15 and performs encoding by selecting necessary tools from these tools (transformer tools A to X, quantizer tools A to X, inverse quantizer tools A to X, and inverse transformer tools A to X in FIG. 5).
A decoder for decoding the coded data shown in FIG. 1B is accomplished by replacing a circuit block 19 constituting the decoder shown in FIG. 3 with a circuit block 22 shown in FIG. 5. That is, this decoder has a plurality of different tools in each of the inverse quantizer 14 and the inverse transformer 15 and performs decoding by selecting necessary tools from these tools (inverse quantizer tools A to X and inverse transformer tools A to X in FIG. 5).
In this decoding process, the tool information shown in FIG. 1B, i.e., the motion compensation tool 1, the inverse transform tool 3, the motion compensation tool 5, the inverse transform tool 7, and the quantization tool 9, are transmitted to a controller 23. The picture information following these tool information, i.e., the motion vector information 2, the transform coefficient 4, the motion vector information 6, and the transform coefficient 8, are transmitted to the respective corresponding tools for processing these picture information. The controller 23 selects tools to be used (from the inverse quantizers A to X and the inverse transformers A to X in FIG. 5) on the basis of the tool information. Each picture data is decoded by using the tools selected by the controller 23.
The method of realizing decoding by executing appropriate software by using a general-purpose arithmetic unit and a compiler will be described below. Decoding of coded data with the structure shown in FIG. 1B will be described with reference to FIG. 6. FIG. 6 is a block diagram showing the structure of a decoder having a general-purpose arithmetic processor 24 and a compiler 25. The tool information shown in FIG. 1B, i.e., the motion compensation tool 1, the inverse transform tool 3, the motion compensation tool 5, the transform tool 7, and the quantization tool 9, are supplied to the compiler 25 which in turn generates a processing program for controlling the operation of the general-purpose arithmetic processor 24. The picture information following these pieces of tool information, i.e., the motion vector information 2, the transform coefficient 4, the motion vector information 6, the transform coefficient 8, and the quantization step 10, are supplied to the general-purpose arithmetic processor 24. In accordance with the processing program generated by the compiler 25, the general-purpose arithmetic processor 24 processes the coded picture information following the tool information, decodes the coded data, and generates decoded data.
Each tool information transmitted from the encoder contains not only information indicating the type of tool but also a tool main body which describes the procedure of the tool. Consequently, even if the decoder is not equipped with necessary tools, the received coded data can be decoded.
If, however, there is a transmission error in transmitting a tool main body, the received tool main body itself does not normally operate in some instances when coded data is decoded by using this transmitted tool main body. Generally, the error rate rises in data transmission using mobile transmission networks. If an error occurs in data, such as information of a tool main body, which describes the procedure itself, not only the data processing of the system is not normally executed but a fatal operation failure sometimes occurs in the whole system.
A general method by which this is prevented is to check whether received data has an error by adding a parity to transmission data. That is, a parity is added in units of certain transmission blocks, and an arithmetic operation is performed for this parity and a data main body. On the basis of the result of the arithmetic operation, whether the received data has an error is determined on the receiver side. If the receiver side determines that there is no error, the receiver side returns data indicating a reception success. If the receiver side determines that there is an error, the receiver side returns data indicating a reception failure. When receiving the data indicating a reception success, the transmitter side transmits the next block. When receiving the data indicating a reception failure, the transmitter side again transmits the current block. This procedure is repeated in units of transmission blocks.
FIG. 7 is a flow chart for explaining an operation algorithm for error determination in conventional decoders. This algorithm is used to perform error determination when a data sequence having the structure shown in FIG. 1B is received.
When coded data is received, whether a tool 1 contained in this coded data has an error is checked (step S1). If it is determined that the tool 1 has an error (No in step S1), a request to retransmit the tool 1 is output to the transmission side (step S2), and the flow waits for retransmitted data. If there is no error in the tool 1 (Yes in step S1), whether a tool 2 has an error is checked (step S3). If it is determined that the tool 2 has an error (No in step S3), a request to retransmit the tool 2 is output to the transmission side (step S4), and the flow waits for retransmitted data. If there is no error in the tool 2 (Yes in step S3), the received coded data is decoded by using these tools 1 and 2 (step S5), and the flow waits until the next coded data is input.
When the next coded data is received, whether a tool 3 contained in this coded data has an error is checked (step S6). If it is determined that the tool 3 has an error (No in step S6), a request to retransmit the tool 3 is output to the transmission side (step S7), and the flow waits for retransmitted data. If there is no error in the tool 3 (Yes in step S6), whether a tool 4 has an error is checked (step S8). If it is determined that the tool 4 has an error (No in step S8), a request to retransmit the tool 4 is output to the transmission side (step S9), and the flow waits for retransmitted data. If there is no error in the tool 4 (Yes in step S8), whether a tool 5 has an error is checked (step S10). If it is determined that the tool 5 has an error (No in step S10), a request to retransmit the tool 5 is output to the transmission side (step S11), and the flow waits for retransmitted data. If there is no error in the tool 5 (Yes in step S10), the received coded data is decoded by using these tools 3, 4 and 5 (step S12), and the flow waits until the next coded data is input.
In the error determination algorithm described above, if even one tool in which an error is detected exists, no decoding processing is executed by using other normal tools. That is, no decoding is performed until no more error is detected in all tools. Also, if an error is detected in a certain transmission block in the tool 1 contained in the received coded data (No in step S1), the retransmission procedure of the tool 1 (step S2) is completed, and error determination is again executed for the retransmitted tool (step S1). Unless it is determined that the tool 1 is normally received, the encoder does not transmit the next transmission block (tool 2).
As a consequence, if the number of retransmission procedures increases, the operating efficiency on the reception side significantly decreases. This also decreases the use efficiency of the communication transmission path. Furthermore, even when a tool is normally transmitted, whether the tool normally operates on the receiver side can be checked by neither the transmitting terminal nor the receiving terminal.
Conventionally, moving pictures are reproduced on general-purpose computers such as personal computers as well as the encoding/decoding apparatus shown in FIGS. 2 and 3. Since a plurality of picture data can be reproduced on a single computer by using a plurality of picture reproducing software, this computer can be regarded as a general-purpose picture reproducing apparatus.
Conventional dedicated picture reproducing apparatuses including the above prior art have a high picture reproducibility. However, to reproduce picture data to be reproduced by different picture reproducing algorithms, it is necessary to prepare different picture reproducing apparatuses corresponding to these picture data.
A general-purpose picture reproducing apparatus is configured such that software of a specific picture reproducing algorithm is executed on general-purpose hardware. Therefore, a plurality of picture reproducing algorithms can be executed on single hardware by preparing software for each of different picture reproducing algorithms. The disadvantage of these general-purpose picture reproducing apparatuses is a lower picture reproducibility than that of dedicated picture reproducing apparatuses.
As described earlier, the picture coding system at a very low bit rate is being actively, internationally standardized as MPEG4 (Moving Picture Experts Group 4). Also, many new coding systems have been developed for picture coding techniques such as a DCT system and a Wavelet transform system. It is considered that the number of coding systems will continue to increase in the future.
The purpose of this MPEG4 is to give a coding system which is versatile and expandable independent of machines. As a consequence, a software download type system is constructed. Accordingly, an MSDL (MPEG4 Syntax Description Language) is constructed which generally describes a picture data encoding/decoding method, a method of transmission of the encoding/decoding method, security in transmission, and a transmission error countermeasure.
FIG. 8 shows the components of the MSDL. A coding system descriptor code which is the description of a picture data encoding/decoding method is constituted by a plurality of profiles. One profile has a plurality of algorithms, and one algorithm has a plurality of tools. Thus the coding system descriptor code has a tree structure, and these tools, configuration methods of the tools, and intertool interfaces are described by the MSDL. Also, in the existing coding system descriptor codes, a new coding system descriptor code can be constituted by the MSDL by exchanging or adding tools.
The MPEG 4 defines Configuration Phase, Learning Phase, and Transmission Phase during communication, and the contents of processing are determined for each phase. In Configuration Phase, to start call setting or establish a communication path, whether an encoder and a decoder can perform appropriate communication is checked. For this purpose, parameters describing, e.g., capability, conditions, and environment are exchanged. Also, after these parameters are exchanged, an environment for communication is determined or whether the control is to proceed on to next Learning Phase is checked. In Learning Phase, to allow the decoder to correctly analyze coded data transmitted from the encoder, transmission of an analysis method itself or construction of the analysis method is designated. In Transmission Phase, coded data is transmitted by the format specified in these two phases.
FIG. 9 shows a conventional picture encoding apparatus of the MPEG4 system. In FIG. 9, reference numeral 31 denotes a communication state controller; 32, a picture encoder; and 33, a coding system descriptor code transmitter. The communication state controller 31 controls the state of communication and, in accordance with the communication state, selects transmission of picture coded data or transmission of a coding system descriptor code. The picture encoder 32 encodes an input picture and forms picture coded data. The coding system descriptor code transmitter 33 transmits tools constituting the coding system used in the picture encoder 32 in order for the terminal on the other end to recognize the coding system. In Learning Phase described above, contacts a and c of a switch are closed and the coding system descriptor code used in the apparatus is transmitted to the partner terminal. When the control proceeds on to Transmission Phase, contacts b and c of the switch are closed and the picture coded data formed by the picture encoder 32 is transmitted.
With this configuration, a software download type system can be constructed. Consequently, any new developed coding system can be processed.
Presently, various coding systems have been developed in addition to the systems described above, and it is difficult to discriminate between them, i.e., it is difficult to choose one of these coding systems. These coding systems have their respective features, and different types of pictures have different appropriate coding systems. Also, cameras for inputting pictures may be switched, or panned or tilted, or the object itself may be changed during communication. In these cases the properties of an input picture largely change. Consequently, it is necessary to switch coding systems during communication in order to transmit clearer pictures in a narrow band. However, in the conventional picture communication systems the coding systems cannot be switched during communication.